Machine learning model modification and natural language processing

ABSTRACT

A system, computer program product, and method are provided to automate a framework for knowledge graph based persistence of data, and to resolve temporal changes and uncertainties in the knowledge graph. Natural language understanding, together with one or more machine learning models (MLMs), is used to extract data from unstructured information, including entities and entity relationships. The extracted data is populated into a knowledge graph. As the KG is subject to change, the KG is used to create new and retrain existing machine learning models (MLMs). Weighting is applied to the populated data in the form of veracity value. Blockchain technology is applied to the populated data to ensure reliability of the data and to provide auditability to assess changes to the data.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation patent application claiming thebenefit of the filing date of U.S. patent application Ser. No.15/866,706 filed on Jan. 10, 2018 and titled “Machine Learning ModelModification and Natural Language Processing”, now pending, the entirecontents of which are hereby incorporated by reference.

BACKGROUND

The present embodiment(s) relate to natural language processing. Morespecifically, the embodiment(s) relate to an artificial intelligenceplatform to convey and utilize recollection in natural languageprocessing.

In the field of artificial intelligent computer systems, naturallanguage systems (such as the IBM Watson® artificial intelligentcomputer system and other natural language question answering systems)process natural language based on knowledge acquired by the system. Toprocess natural language, the system may be trained with data derivedfrom a database or corpus of knowledge, but the resulting outcome can beincorrect or inaccurate for a variety of reasons relating to thepeculiarities of language constructs and human reasoning or new trainingdata that is incorrect.

Machine learning, which is a subset of Artificial intelligence (AI),utilizes algorithms to learn from data and create foresights based onthis data. AI refers to the intelligence when machines, based oninformation, are able to make decisions, which maximizes the chance ofsuccess in a given topic. More specifically, AI is able to learn from adata set to solve problems and provide relevant recommendations. AI is asubset of cognitive computing, which refers to systems that learn atscale, reason with purpose, and naturally interact with humans.Cognitive computing is a mixture of computer science and cognitivescience. Cognitive computing utilizes self-teaching algorithms that usedata minimum, visual recognition, and natural language processing tosolve problems and optimize human processes.

Cognitive systems are inherently non-deterministic. Specifically, dataoutput from cognitive systems are susceptible to information providedand used as input. For example, as new machine learning models aredeployed there is not guarantee that the system will extract the sameentities as done previously. New models may adversely affect prior modelresults. Similarly, an error introduced through a document may result inextracting incorrect data and providing the incorrect data as output.Accordingly, there is a need to create deterministic behavior in thecognitive system(s).

SUMMARY

The embodiments include a system, computer program product, and methodfor natural language processing directed at deterministic data forcognitive systems.

In one aspect, a system is provided with a processing unit operativelycoupled to memory, with an artificial intelligence platform incommunication with the processing unit and memory. A knowledge engine,which is in communication with the processing unit, is provided to traina machine learning model (MLM). More specifically, the knowledge enginereceives natural language input to query against the first knowledgegraph for extraction of a triplet, and employs the identified MLM to asecond knowledge graph for extraction of one or more triplets. Eachtriplet includes a subject, object, and a relationship. For eachtriplet, a BC identifier is obtained and a triplet veracity value isidentified in a corresponding BC ledger. The knowledge engine detects amodification by comparison of the extracted triplets, and evaluates thedetected modification, including employing the obtained BC identifier toassess veracity of the modification.

In another aspect a computer program product is provided to processnatural language. The computer program product includes a computerreadable storage device having embodied program code that is executableby a processing unit. Program code is provided to receive naturallanguage input, query the input against the first knowledge graph, andextract one or more triplets from the first knowledge graph. Inaddition, program code is provided to apply the selected MLM to a secondknowledge graph and extract one or more triplets from the secondknowledge graph. Each triplet includes a subject, object, and arelationship there between. For each extracted triplet, the program codeobtains a BC identifier and identifies a triplet veracity value from acorresponding BC ledger. The program code detects a modification of thefirst knowledge graph based on the extracted triplets, and evaluates thedetected modification.

In yet another aspect, a method is provided for processing naturallanguage. The method includes receiving natural language input, queryingthe input against a first knowledge graph, and extracting one or moretriplets from the first knowledge graph. A selected MLM is applied to asecond knowledge graph that is different from the first knowledge graph.One or more triplets are extracted from the second knowledge graph. Eachextracted triplet includes a subject, object, and an associatedrelationship. For each extracted triplet, a BC identifier associatedwith the triplet is obtained, and a triplet veracity value from acorresponding BC ledger is identified. Following application of the MLM,a modification of the first knowledge graph is detected, with themodification being content and/or structural change, and the detectedmodification is evaluated, which includes employing the BC identifier toassess veracity of the modification.

These and other features and advantages will become apparent from thefollowing detailed description of the presently preferred embodiment(s),taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The drawings reference herein forms a part of the specification.Features shown in the drawings are meant as illustrative of only someembodiments, and not of all embodiments, unless otherwise explicitlyindicated.

FIG. 1 depicts a system diagram illustrating a schematic diagram of anatural language process system.

FIG. 2 depicts a block diagram a block diagram is provided illustratingthe NL processing tools shown in FIG. 1 and their associated APIs.

FIG. 3 depicts a flow chart illustrating a process for populating aknowledge graph (KG) from natural language (NL) output of a naturallanguage processing (NLP) system.

FIG. 4 depicts a flow chart illustrating a process for creating a newtriplet from extracted data.

FIGS. 5A and 5B depict a flow chart illustrating a process forextracting triplets from NLP output.

FIG. 6 depicts a flow chart illustrating a process for partitioning aKG.

FIG. 7 depicts a flow chart illustrating a process for linking two KGs.

FIGS. 8A and 8B depict a flow chart illustrating a process forleveraging a machine learning model (MLM) to augment the query input.

FIG. 9 depicts a flow chart illustrating a process for training anexisting MLM.

FIG. 10 depicts a flowchart illustrating a process for progressive andadaptive MLM configuration.

DETAILED DESCRIPTION

It will be readily understood that the components of the presentembodiments, as generally described and illustrated in the Figuresherein, may be arranged and designed in a wide variety of differentconfigurations. Thus, the following details description of theembodiments of the apparatus, system, method, and computer programproduct of the present embodiments, as presented in the Figures, is notintended to limit the scope of the embodiments, as claimed, but ismerely representative of selected embodiments.

Reference throughout this specification to “a select embodiment,” “oneembodiment,” or “an embodiment” means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiments. Thus, appearances of thephrases “a select embodiment,” “in one embodiment,” or “in anembodiment” in various places throughout this specification are notnecessarily referring to the same embodiment.

The illustrated embodiments will be best understood by reference to thedrawings, wherein like parts are designated by like numerals throughout.The following description is intended only by way of example, and simplyillustrates certain selected embodiments of devices, systems, andprocesses that are consistent with the embodiments as claimed herein.

Ontology functions as a structural framework to organize information andconcepts. Natural language understanding (NLU) is a subset of naturallanguage processing (NLP). NLU uses algorithms to transform speech intoa structured ontology. In one embodiment, the ontology is constructedfrom the taxonomy of NLU output. NLU provides the definitions requiredto construct the ontology in terms of classes, subclasses, domain,range, data properties, and object properties. Ontology individuals aremapped to objects. Processing the same or similar documents provides thedata required to build the ontology, also referred to as the initialontology. The ontology is defined by the Machine Learning Model (MLM)being applied by a knowledge graph (KG) manager to a data store; theontology is constructed using output of an associated NLP service. Morespecifically, the ontology is generated with facts or mentions that theMLM has generated. The facts or mentions make up the individuals of theontology. In one embodiment, the ontology is in the form of a KG withthe facts or mentions represented as nodes in the graph. The structureof the KG may be kept constant, while allowing information to be addedor removed. Similarly, the ontology may be used to create new andretrain existing MLMs. In one embodiment, when the KG is modified, newentities and relationships are realized and employed to automatetraining of the MLM; the MLM becomes dynamic and progressive.Accordingly, the ontology as represented by the KG and the MLM areinter-related.

Referring to FIG. 1, a schematic diagram of a natural languageprocessing system (100) is depicted. As shown, a server (110) isprovided in communication with a plurality of computing devices (180),(182), (184), (186), and (188) across a network connection (105). Theserver (110) is configured with a processing unit (112) operativelycoupled to memory (114) across a bus (116). A tool in the form of aknowledge engine (170) is shown local to the server (110), andoperatively coupled to the processing unit (112) and/or memory (114). Asshown, the knowledge engine (170) contains one or more tools(172)-(178). The tools (172)-(178) provide natural language processingover the network (105) from one or more computing devices (180), (182),(184), (186) and (188). More specifically, the computing devices (180),(182), (184), (186), and (188) communicate with each other and withother devices or components via one or more wires and/or wireless datacommunication links, where each communication link may comprise one ormore of wires, routers, switches, transmitters, receivers, or the like.In this networked arrangement, the server (110) and the networkconnection (105) may enable natural language processing and resolutionfor one or more content users. Other embodiments of the server (110) maybe used with components, systems, sub-systems, and/or devices other thanthose that are depicted herein.

The tools, including the knowledge engine (170), or in one embodiment,the tools embedded therein including the KG manager (172), the accuracymanager (174), the BC manager (176), and the MLM manager (178), may beconfigured to receive input from various sources, including but notlimited to input from the network (105), one or more knowledge graphsfrom a node-graph data store (160) operatively coupled to a corpus ofstructured data (168) via an interface (166), a BC network (150), and alibrary of one or more machine learning models (MLMs) (140). As shown,the node-graph data store (160) functions as a library (162) ofknowledge graphs, with a plurality of KGs, including KG₀ (164A), KG₁(164B), and KG_(N) (164N). The quantity of KGs shown herein should notbe considered limiting. Each KG is a representation of an ontology ofconcepts. More specifically, each KG (162), (164), and (166) includes aplurality of related subjects and objects. In one embodiment, relatedKGs are stored in an associated KG container, with the corpus (160)storing one or more KG containers. In one embodiment, KGs may also beacquired from other sources, and as such, the data store depicted shouldnot be considered limiting.

The various computing devices (180), (182), (184), (186), and (188) incommunication with the network (105) demonstrate access points forcontent creators and content uses. Some of the computing devices mayinclude devices for a database storing the corpus of data as a body ofinformation used by the knowledge engine (170), and in one embodimentthe tools (172)-(178), to embed deterministic behavior into the system.The network (105) may include local network connections and remoteconnections in various embodiments, such that the knowledge engine (170)and the embedded tools (172)-(178) may operate in environments of anysize, including local and global, e.g. the Internet. Additionally, theserver (110) and the knowledge engine (170) serves as a front-end systemthat can make available a variety of knowledge extracted from orrepresented in documents, network accessible sources and/or structureddata sources. In this manner, some processes populate the server (110)with the server (110) also including input interfaces to receiverequests and respond accordingly. Content creators and content users mayalso be available in data repositories, such as, but not limited to,(140) and (160), and the list of demonstrated access points here shouldnot be considered limiting.

As shown, the node-graph data store (160) is operatively coupled to theserver (110). The node-graph data store (160) includes a KG library(162) with one or more KGs (164A)-(164N) for use by the server (110).Content users may access the system via API administration ororchestration platforms, as shown and described in FIG. 2, and naturallanguage input received via the NLU input path.

As described in detail below, the server (110) and the knowledge engine(170) processes natural language queries through use of one or moremachine learning models, hereinafter MLMs, to extract or store contentin one or more KGs stored in the node-graph data store (160). Blockchaintechnology, hereinafter “BC”, is leveraged into the content toeffectively provide authenticity, e.g. provenance, of stored or receiveddata. The MLM manager (178) functions as a tool, or in one embodiment,an API within the knowledge engine (170), and is used to create, link,and/or modify an associated MLM. As further described below, MLMs aregenerated, created, or modified specific to a particular knowledgedomain. The MLMs are created to extract entities and relationships fromunstructured data. These models are specifically created to understand aparticular domain of knowledge (e.g. biographical information, stockmarket, astronomy, etc.).

The BC is represented herein as the BC Network (150) in the form of adecentralized and distributed digital ledger for recording the historyof transactions. More specifically, the BC refers to a type of datastructure that enables digitally identifying and tracking transactionsand sharing this information across a distributed network of computers.The BC effectively creates a distributed trust network via transparentlyand securely tracking ownership. As shown and described herein, the BCis leveraged together with the MLM manager (178), accuracy manager(174), and KG manager (172) to integrate knowledge with natural languageprocessing.

The server (110) may be the IBM Watson® system available fromInternational Business Machines Corporation of Armonk, N.Y., which isaugmented with the mechanisms of the illustrative embodiments describedhereafter. The IBM Watson® knowledge manager system imports knowledgeinto natural language processing (NLP). Specifically, as described indetail below, as data is received, organized, and/or stored, the datamay be true or false. The server (110) alone cannot differentiate, ormore specifically, authenticate the veracity of the data. As shownherein, the server (110) receives input content (102) which it thenevaluates to extract features of the content (102) that in turn are thenapplied to the node-graph data store (160). In particular, receivedcontent (102) may be processed by the IBM Watson® server (110) whichperforms analysis to evaluate or impart authenticity of the inputcontent (102) using one or more reasoning algorithms.

To process natural language, the server (110) utilizes an informationhandling system in the form of the knowledge engine (170) and associatedtool (172)-(178) to support NLP. Though shown as being embodied in orintegrated with the server (110), the information handling system may beimplemented in a separate computing system (e.g., 190) that is connectedacross the network (105) to the server (110). Wherever embodied, one ormore MLMs are utilized to manage and process data, and morespecifically, to detect and identify natural language and create orutilize deterministic output. As shown, the tools include the KG manager(172), the accuracy manager (174), a BC manager (176) and the MLMmanager (178). The MLM manager (178) is shown operatively coupled to anMLM library (140) shown herein with a plurality of MLMs, including MLM₀,(142), MLM₁ (144), and MLM_(N) (146), although the quantity of MLMsshown and described should not be considered limiting. It is understoodthat in one embodiment the MLM is an algorithm employed or adapted tosupport the NLP. Although shown local to the server (110), tools(170)-(178) may collectively or individually be embedded in memory(114).

One or more MLMs (142)-(146) function to manage data, including storedata in a KG. As understood, a KG is a structured ontology and does notmerely store data. Specifically, the knowledge engine (170) extractsdata and one or more data relationships from unstructured data, createsan entry for the extracted data and data relationship(s) in a KG, andstores the data and data relationship(s) in the KG entry. In oneembodiment, data in the KG is stored or represented in a node and arelationship between two data elements is represented as an edgeconnecting two nodes. Similarly, in one embodiment, each node has a nodelevel veracity value and each relationship has a relationship veracityvalue, with the relationship veracity value calculated based on theveracity values of the two inter-connected nodes. In addition to thedata extraction and storage, the MLM(s), MLM₀, (142) assigns orotherwise designates a veracity value to the data stored in the KG. Inone embodiment, the veracity value is a composite score comprised ofstaunchness, source reliability, and human feedback, as described indetail below. In one embodiment, the veracity value may includeadditional factors or a subset of the factors, and as such, should notbe considered limiting. The assigned veracity value is stored in the KG.The assigned veracity value is also stored in an entry in the identifiedBC ledger. Each entry in the BC ledger has a corresponding identifier,referred to herein as a BC identifier, which identifies the ledger andaddress of the ledger entry. The BC identifier is stored in the KG withthe identified data and identifies the corresponding BC ledger andlocation of the stored veracity value. In one embodiment, the KG manager(172) manages storage of the BC identifier in the KG. Accordingly, theassigned or created veracity value is stored in the BC and is aduplicated copy of the veracity values in the KG in the node-graph datastore (160).

It is understood that each KG organizes and provides structure to largequantities of data. A KG may be a single ontology, or in one embodiment,a KG or a KG container may be comprised of a plurality of KGs that arelinked together to demonstrate their relationship or association. The KGmanager (172) functions to manage structure and organization of the KGs.For example, a large KG may be too cumbersome or expensive to manage. Inthis scenario, the KG manager (172) may partition the KG, effectivelycreating at least two partitions, e.g. a first KG partition and a secondKG partition. The KG may be partitioned based on one or more factors.For example, in one embodiment, the KG may be partitioned by topics orsub-topics. Similarly, each fact represented in the KG has an associatedveracity value that is a composite of a plurality of factors, includingbut not limited to a staunchness indicator, a source reliabilitymeasure, and a human feedback factor. The KG manager (172) may partitionthe KG based upon the veracity value, or in one embodiment, based on oneor more of the factors that comprise the veracity value. In oneembodiment, after the KG has been partitioned into at least a first anda second partition, the KG manager (172) may designate one or more ofthe components of the veracity value to each node or edge represented inthe partition. For example, following the KG partitioning, the KGmanager (172) may populate and assign a first reliability value to datain the first partition, and in one embodiment, the KG manager (172) mayfurther populate and assign a second reliability value, different thanthe first reliability value, to data in the second partition.Modification of one or more of the components of the veracity valueeffectively changes the veracity value. However, it is understood thatthe values of one or more of the components of the veracity value maychange over time, and as such, this change is reflected or embodied withthe associated data. Accordingly, the KG manager (172) functions tomanage data and provide structure and value to the data.

One of the functions of the KG manager (172) is to link or join two ormore KGs. Joining or linking KGs is the inverse of partitioning a KG.The functionality of joining or linking KGs requires the KG manager(172) to compare one or more data elements in one KG with one or moredata elements in a second KG and to eliminate or at least reduce theappearance of duplicate data. As articulated above, each data elementrepresented in the KG has an associated composite score. The KG manager(172) may use one component, multiple components, or the veracity valueitself as a factor for the data comparison and evaluation. Once joinedor linked, it may be feasible or warranted to remove duplicate dataitems. Data in the linked KGs determined to be duplicate data isselectively removed by the KG manager (172). One characteristic ofremoving duplicate data is the ability to maintain a constant structureof the KG. Accordingly, the KG manager (172) functions to manage thestructure of the KG by managing the data represented in the KG.

The BC manager (176) has multiple functions with respect to the machinelearning environment. As described above, the BC manager (176) mayfunction with the MLM(s) to maintain the authenticity of the associateddata. The BC manager (176) produces contracts for BC networkinteractions, provides provenance, retrieves BC information, and managesall BC interactions for the system.

The evaluation of the NL input is managed by the MLM, MLM₀, (142). Aconflict or error associated with the NL input is identified by queryresults from the KG generated from the NL input, and more specificallythe sorting of the query results. When there is a conflict between thequery results and the NL input, with the query results having a strongveracity value, it is an indication that the NL input may be incorrect.The accuracy manager (174) corrects the NL input by replacing thelanguage of the NL input with an identified or selected triplet from thegenerated list. The triplet, also referred to herein as memory, is basedon two or more nodes in the KG and a relationship between the two ormore nodes. In one embodiment, the triplet is a subject-verb-objectrelationship as captured from the KG. In one embodiment, theidentification or selection may be based on the highest veracity value,which in one embodiment is selected by the user. Similarly, in anotherembodiment, the identification or selection may be based on one or moreof the factors that comprise the composite veracity value. Another formof a conflict may arise when the knowledge engine (150) identifies animmutable factor associated with one or more entries in the list andfurther identifies a conflict between the immutable factor and the NLinput. This conflict is resolved by the accuracy manager (174)correcting the NL input with a replacement of the language of the NLinput with the triplet associated with the entry having the immutablefactor. In addition to a conflict, another resolution may be in theaccuracy manager (174) identifying a partial match between the NL inputwith the sorted list entries. The partial match enables or directs theKG manager (172) and the BC manager (176) to create a new entry in theKG and a corresponding BC ledger, respectively, for the NL input. Inaddition, the KG manager (172) connects the new entry and the existingKG entry corresponding to the partial match. It is further understoodthat the NL input may not generate any matches, e.g. empty set. If thereis no match, the KG manager (172) and the BC manager (176) create a newKG entry and BC ledger entry, respectively, corresponding to the NLinput. Accordingly, NL input is processed by the MLM, MLM₀, (142), andin one embodiment the accuracy manager (174), in view of the dataorganized in the KG(s).

As shown and described herein, an MLM library (140) is operativelycoupled to the server (110) and contains a plurality of MLMs to supportnatural language processing in the AI platform. One or more of the MLMsmay be dynamic and trained to adapt to new entities and relationships.Different KGs may be associated with different knowledge domains. Forexample, a first MLM, MLM₀ (142), may be identified or selected from thelibrary (140) based on its alignment with KG₀ (164A). In response toprocessing NL input, MLM₀ (142) may be applied against KG₀ (164A) andseparately applied against a second KG, KG₁ (164B). The MLM manager(178) processes results from both KGs together with their correspondingveracity values, and based on the processing, a modification of one ofthe KGs is identified. In one embodiment, the veracity values areevaluated to identify the authenticity of the modification. Subject tothe authentication, the MLM manager (178) dynamically modifies theassociated MLM, MLM₀ (142). In one embodiment, the identifiedmodification may be an expansion of the associated data set to includean additional field. Similarly, in one embodiment, the MLM manager (178)may ascertain that the modification is synchronic or diachronic, and usethis classification as an element to oversee the modification. In oneembodiment, the modification of MLM₀ (142) results in creation of a newMLM, e.g. MLM_(N) (146), and in one embodiment retention of the originalMLM, MLM₀ (142). Accordingly, the MLM library (140) may expand subjectto the dynamic modification of the MLMs.

Types of information handling systems that can utilize system (110)range from small handheld devices, such as a handheld computer/mobiletelephone (180) to large mainframe systems, such as a mainframe computer(182). Examples of a handheld computer (180) include personal digitalassistants (PDAs), personal entertainment devices, such as MP4 players,portable televisions, and compact disc players. Other examples ofinformation handling systems include pen or tablet computer (184),laptop or notebook computer (186), personal computer system (188) andserver (190). As shown, the various information handling systems can benetworked together using computer network (105). Types of computernetwork (105) that can be used to interconnect the various informationhandling systems include Local Area Networks (LANs), Wireless Local AreaNetworks (WLANs), the Internet, the Public Switched Telephone Network(PSTN), other wireless networks, and any other network topology that canbe used to interconnect the information handling systems. Many of theinformation handling systems include nonvolatile data stores, such ashard drives and/or nonvolatile memory. Some of the information handlingsystems may use separate nonvolatile data stores (e.g., server (190)utilizes nonvolatile data store (190 a), and mainframe computer (182)utilizes nonvolatile data store (182 a). The nonvolatile data store (182a) can be a component that is external to the various informationhandling systems or can be internal to one of the information handlingsystems.

An information handling system may take many forms, some of which areshown in FIG. 1. For example, an information handling system may takethe form of a desktop, server, portable, laptop, notebook, or other formfactor computer or data processing system. In addition, an informationhandling system may take other form factors such as a personal digitalassistant (PDA), a gaming device, ATM machine, a portable telephonedevice, a communication device or other devices that include a processorand memory.

An Application Program Interface (API) is understood in the art as asoftware intermediary between two or more applications. With respect NLprocessing system shown and described in FIG. 1, one or more APIs may beutilized to support one or more of the tools (172)-(178) and theirassociated functionality. Referring to FIG. 2, a block diagram (200) isprovided illustrating the NL processing tools and their associated APIs.As shown, a plurality of tools are embedded within the knowledge engine(205), with the tools including the accuracy manager (210) associatedwith API₀ (212), the KG manager (220) associated with API₁ (222), the BCmanager (230) associated with API₂ (232), and the MLM manager (240)associated with API₃ (242). Each of the APIs may be implemented in oneor more languages and interface specifications. API₀ (212) provides forasset comparison, veracity determination, veracity decision, andveracity assignments; API₁ (222) provides for KG creation, update, anddeletion; API₂ (232) provides MLM creation, update and deletion; andAPI₃ (242) provides for BC contract creation, block creation, networkcommunication, and block addition. As shown, each of the APIs (212),(222), (232), and (242) are operatively coupled to an API orchestrator(250), otherwise known as an orchestration layer, which is understood inthe art to function as an abstraction layer to transparently threadtogether the separate APIs. In one embodiment, the functionality of theseparate APIs may be joined or combined. As such, the configuration ofthe APIs shown herein should not be considered limiting. Accordingly, asshown herein, the functionality of the tools may be embodied orsupported by their respective APIs.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 3which illustrates a process in a form for initializing a KG. When asystem is initialized, the KG is empty. MLMs are created or utilized toextract entities and relationships from unstructured data. The MLMs arecreated to understand a particular domain of knowledge, i.e.biographical information, financial market, fields of science, etc.Representative data is used to teach the system to identify the text ofthe entities and relationships defined in the model. Referring to FIG.3, a flow chart (300) is provided illustrating a process for populatinga KG from natural language output of an NLP system. As part of the KGinitialization and population process, veracity values for the extractedtriplets are specified. The veracity value is comprised of a staunchnessindicator, a source reliability indicator, and a human feedbackindicator. In one embodiment, each of the indicators that comprise theveracity value is a numerical value on a scale between 0 and 1. Thestaunchness indicator reflects certainty of an underlying fact. In oneembodiment, a staunchness value of 1 reflects that the fact isdefinitely true, a value of 0 reflects that the fact is definitelyfalse, and a value between 0 and 1 indicates a level of certainty oruncertainty about the fact. The source reliability factor is associatedwith the source, e.g. origin, of the fact, including but not limited to,data and time a fact was ascertained. The human feedback indicatortracks the quantity of affirmations and refutations of a fact. In oneembodiment, this factor tracks the quantity of responses. Accordingly,as the KG is initialized and populated with data, the components of theveracity value are selected or set to be assigned to the tripletsextracted via the NLP system.

Classification, such as synchronic and diachronic information, isemployed to describe data that either stays constant or can change overtime, respectively. In the case of an example of supervised training,the staunchness value is set to 1, the source reliability value is setto 1, and the human feedback is set to 0. These values are merely anexample, and in one embodiment may vary. In one embodiment, a KGapplication program interface (API) provides a platform to specify theveracity values. As shown, an MLM is created by defining entities andrelationships (302). The MLM is trained using representative data (304).Following step (304), the MLM is used with NLP to extract triplets fromthe training data (306). The extracted triplet may be saved to a file orstreamed. In one embodiment, the extracted triplet is asubject-verb-object relationship. Following step (306), the extractedtriplets are employed to populate the KG (308). In one embodiment, theKG API is used to read and parse out the triplets from the NLU output.In one embodiment, the triplets populated into the KG are referred to asmemories. The MLM is created through training, after which the MLM isapplied against data to populate the KG. Accordingly, the MLM togetherwith the NLP extract triplets from data and populate a previously emptyKG.

For each subject-entity extracted from the NLP output (310), it isdetermined if the subject-entity exists in the associated KG (312). Apositive response to the determination at step (312) is followed bydetermining if there is a known relationship associated with theextracted subject-entity (314). If the response to the determination atstep (314) is affirmative, then it is determined if the subject-entitytogether with the associated relationship and the assigned veracityvalue is presented in the KG (316). A positive response to thedetermination at step (316) is an indication that the subject-entityrelationship is present in the KG, and the process concludes. However, anegative response to any one of the determinations shown at steps (312),(314), and (316), is followed by the creation of a new triplet and anentry of the new triplet in the KG (318). Accordingly, as shown, the MLMis employed to extract data from an NLP document and to access the KGmanager to selectively populate the KG with the extracted data.

Referring to FIG. 4, a flow chart (400) is provided illustrating aprocess for creating a new triplet from the extracted data. As describedin FIG. 3, the veracity value components of the extracted data areestablished or assigned. In one embodiment, the veracity valuecomponents are established based on the supervision associated with theKG initialization. For each new triplet, e.g. subject-verb-objectrelationship, the veracity values are assigned to the triplet (402). Inone embodiment, the veracity values are assigned via the KG API.Following step (402), an entry is created in a corresponding ordesignated BC ledger (404). More specifically, at step (404), the BCentry stores the triplet veracity values and an identifier, referred toherein as a BC identifier, is created and thereafter retrieved. In oneembodiment, the retrieved BC identifier is a uniform resource identifier(URI) or other unique asset identifier. Following step (404), the newtriplet is inserted into the KG together with the associated BCidentifier (406). In one embodiment, the KG API implements the insertionof the triplet and the associated BC identifier at step (406).Accordingly, as demonstrated, the veracity values of each new tripletare stored in a corresponding BC ledger, and an associated BC identifieris stored or otherwise associated with the triplet in the KG entry.

The processes shown and described in FIGS. 3 and 4 may also be employedfor populating a KG from natural language output of an NLP system usingunsupervised training, e.g. the data may be unreliable, or usingsupervised training. As shown and described in FIGS. 3 and 4, the KG APIis utilized to set the veracity values for the data being extracted fromthe NLP output. Depending on the source, the veracity values can be setto indicate uncertainty For example, in one embodiment, the staunchnessindicator may be set to 0.5, the source reliability may be set to 0.5,and the human feedback value may be set to 0. Accordingly, theunsupervised training may be reflected in a different set of veracityvalues.

In the course of processing non-training data, if an exact triplet matchis not found, a new memory is created and stored in the corresponding oridentified KG. This can be realized when considering the processing ofmultiple documents on the same subject. For example, one document mayidentify a fact with a first date, and a second document may identifythe same fact with a second date. However, only one of the dates isfactually correct. As shown in FIGS. 3 and 4, each triplet entered intothe KG has a corresponding veracity value, which functions as anindicator of correctness of the stored memory. These veracity scores maybe used to establish accuracy and/or correctness of conflicting factspopulated into the KG.

Referring to FIGS. 5A and 5B, a flow chart (500) is providedillustrating a process for extracting triplets from the NLP output. Asshown, a query or statement is presented to the KG (502) through theaccuracy manager. The presentation may be for a variety of reasons,including but not limited to, fact checking. The MLM is used with NLP toextract triplets from the KG (504), and the KG API is used to read andparse out triplets from the NLP output (506). The following tableillustrate an example triplet:

TABLE 1 Subject-Entity Relationship Subject-Entity-Value GeorgeWashington Born On Feb. 22, 1832

Following step (506), the variable X_(Total) is assigned to the quantityof parsed triplets (508). It is then determined if X_(Total) is greaterthan zero (510). A negative response to the determination at step (510)concludes the extraction process (512), since this is an indication thatthe query produced an empty set. However, a positive response to thedetermination at step (510) is followed by processing the parsedtriplet(s) (514). The triplet counting variable is set to one (516), andfor each triplet_(X) the KG is queried to fetch all triplets with thesame subject-entity and relationship (518). As shown and described inFIGS. 3 and 4, each triplet has an associated BC identifier. The BCidentifier is utilized to access the corresponding BC ledger and fetchthe stored triplet veracity value (520). Following step (520), thetriplet counting variable is incremented (522). It is then determined ifeach of the identified triplets has been processed (527). A negativeresponse to the determination at step (522) is followed by a return tostep (518). Similarly, a positive response to the determinationconcludes the process of querying the KG and the corresponding BC ledgerentries (526), and the extracted and processed triplets are subject to asort (528). The sort at (528) functions to place the triplets into anorder. For example, in one embodiment, the triplets may be sorted inascending order by the staunchness indicator, source reliability, andhuman feedback. Similarly, the sort order may be customizable toaccommodate specific use cases. For example, in one embodiment, thehuman feedback indicator may be prioritized. Accordingly, the tripletextraction utilizes the KG to obtain or identify the triplet andassociated BC identifier, which is utilized to acquire the associatedveracity values, which are then employed as a characteristic for sortingthe triplets.

The following table, Table 2, is an expansion of Table 1, showing anexample sort of two triplets:

TABLE 2 Subject- Source Human Subject- Relation- Entity StaunchnessReliability Feedback Entity ship Value Indicator Indicator IndicatorGeorge BornOn Feb. 22, 1.0 1.0 0 Washington 1732 George BornOn Feb. 22,0.5 0.5 0 Washington 1832In the example of Table 2, there are two triplet entries, eachassociated with a different subject-entity value. As shown, the entriesare sorted in ascending order by either the staunchness indicator or thesource reliability indicator. The sorting factor should not beconsidered limiting. In one embodiment, the sorting may be inverted andin descending order, or based on a different component of the veracityvalue. The first triplet entry in this example is defined by the subjectentity and the relationship is considered to have the greatest veracityvalue, e.g. veracity score.

The business use case drives the interpretation of the query results.For example, if a triplet with a higher confidence score is realized,the system can be configured to automatically replace the original valueof the subject entity value with a value that has a higher veracityscore. A staunchness indicator is a reflection of the accuracy of thereturned information. As shown, following step (528), the business usecase is applied to the search results (530). The application at step(530) is followed by querying both the KG and the appropriate oridentified BC ledgers (532) associated with the corresponding BCidentifiers in the KG. The query at step (532) obtains all associatedrelationships and subject-entity values. More specifically, this enablesan analytical review of all data for a subject-entity. Following step(532), the NLP input or output data is augmented (534). Examples of theaugmentation include, but are not limited to: correction, analysis,augmentation, and masking. Correction includes replacing a subjectentity value with data from memory. In one embodiment, the replacementis local, e.g. against the query, and is not reflected in the KG or theBC. Analysis includes addition of a list of subject-relation-values withveracity. Augmentation includes supplementing results with all knownsubject-relation values that have the highest level of confidence, e.g.one value per subject-relationship pair. Masking includes deletion ofone or more triplets from the NLP output. Following step (532), theaugmented data is returned. Accordingly, different use cases areoptionally available to drive the interpretation of the search results,which may also be augmented, to return one or more appropriate dataelement from the NLP input.

As shown and described in FIGS. 5A and 5B, one or more queries may beprocessed against a created KG. It is understood that the KGs functionas a tool to organize data, with each triplet reflected in the graphrepresented or otherwise associated with the veracity score components,e.g. staunchness, reliability, and feedback. It is understood that oneor more of the veracity score components may be dynamic, e.g. the valuesare subject to change over time. This change may be uniform throughout aselect KG, thereby affecting each triplet represented in the KG, or thechange may be non-uniform and selectively affecting one or more tripletsin the KG.

Referring to FIG. 6, a flow chart (600) is provided illustrating aprocess for partitioning one or more KGs. The example of thepartitioning shown herein is based on a change in a reliability factor.This is merely an example, and in one embodiment the partitioning may bebased on a change of the staunchness or feedback factors. Thereliability factor reflects a measurement of the reliability of thesource of the data. A reliability factor value is received (602). In oneembodiment, the reliability factor value is part of the NL input andfeedback through the KG API. The KG is consulted to identify entriesassociated with the received reliability value (604). It is thendetermined if any KG entries have been identified (606). A negativeresponse to the determination at step (606) concludes the partitioningprocess since there is no basis for subjecting the KG based on thereceived reliability factor (616). However, a positive response to thedetermination at step (606) is following by creating a partition withinthe KG (608) and populating the created partition with the entries inthe KG with the identified reliability value (610). The partitioncreation at step (608) effectively creates a second partition (612)populated with the remaining entries in the original KG.

It is understood that the entries in the first and second partitions ofthe KG have different reliability factor values. As described above, theveracity value functions as a composite of the staunchness, reliability,and feedback values. A change in any of the individual component valueshas an effect on the composite, which may affect any query results.Following step (612), a veracity evaluation within the KG, including thefirst and second partitions, takes place (614). The evaluation at step(614) includes a comparison of data populated in the first KG partition,e.g. first data, with data populated in the second partition, e.g.second data. In one embodiment, the veracity evaluation is automaticallyperformed after the partitioning. It is understood that data populatedin the first partition will have a different veracity value than thedata in the second partition. The partitioning shown herein is based ona change of one component represented in the veracity value. In oneembodiment, the partitioning may take place on two or more veracityvalue components or changes to the components. Accordingly, a change inany one of the components that comprise the veracity value may includecreation of one or more partitions of the associated KG.

As shown in FIG. 6, a KG may be subject to partitioning. An inverseconcept may take place by linking or otherwise joining two or more KGsand associated BC ledger(s). Referring to FIG. 7, a flow chart (700) isprovided illustrating a process for linking two KGs and the associatedBC ledger(s). In one embodiment, KGs that are at least tangentiallyrelated may be linked. The relationship may be based on content orrelationships represented in the KGs. As shown, a query is presented toa knowledge base (702), and two or more KGs are identified (704). In oneembodiment, the KG API identifies that the two KGs contain data that arerelevant to the query. Similarly, in one embodiment, the KG API mayidentify more than two KGs, and as such, the quantity of identified KGsshould not be considered limiting. A link is established between oramong the identified KGs (706). The linking of two or more KGs maintainsthe structure of the separate KGs, i.e. the structures remain constant.

It is understood that the relationship between the KGs, andspecifically, the data represented therein may provide query resultswith conflicting triplets, e.g. memories. To resolve the potentialconflicts, an evaluation of the linked KGs is conducted to compare dataelements (708). More specifically, the comparison includes an evaluationof data represented in each of the linked KGs (710), including theircorresponding veracity value components. Identified conflicting dataelements are selectively replaced based on at least one of the veracityvalue components (712), e.g. staunchness, reliability, and feedback. Thereplacement follows the structure of the separate KGs. In other words,nodes in the KGs are not removed or added with the linking. Rather thedata represented in the identified nodes may be replaced. Accordingly,the replacement of conflicting entries in the linked KGs mitigatesconflicting query results.

Referring to FIGS. 8A and 8B, a flow chart (800) is providedillustrating leveraging the MLM to augment the query input. Morespecifically, the results of the query submission may indicate an errordirected at the query input. As shown, natural language input isreceived and processed (802). The received input is queried againstcontext (804), including one or more specified KGs, and in oneembodiment, a corresponding BC ledger. The query processing producesresults in the form of one or more triplets, e.g. memories, which areextracted or identified from the specified KG(s) (806). As describedabove, each triplet includes a subject, object, and an associatedrelationship. The variable X_(Total) is assigned to the quantity oftriplets extracted or identified from the KG (808). It is thendetermined if the quantity extracted at step (808) includes at least onetriplet (810). A positive response to the determination at step (810) isfollowed by initializing an associated triplet counting variable (812).Each triplet has a BC identifier that corresponds to a BC ledger entry,which includes the veracity values associated with or assigned to thetriplet. For each extracted or identified triplet, e.g. triplet_(X), theBC identifier is obtained (814), from which the BC ledger is consultedand the corresponding veracity value is identified (816). Following step(816) the triplet counting variable is incremented (818), and anassessment is conducted to determine if each of the extracted oridentified KGs have been evaluated (820). A negative response to thedetermination at step (820) is followed by a return to step (814), and apositive response concludes the triplet extraction and identifiedprocess. Accordingly, for each triplet determined to be associated withthe query input, the associated veracity values are identified.

A negative response to the determination at step (810) is followed bycreating a new triplet for entry in the associated KG (822). The newtriplet corresponds to the received natural language input, e.g. querysubmission, and a veracity score is assigned to the new triplet (824).In addition, an entry in a BC ledger corresponding to the KG is created(826). A BC identifier associated with the BC ledger entry is createdand stored with the new triplet in the KG (828) and the assignedveracity score is stored in the corresponding ledger entry (830).Accordingly, an empty set of triplets returned from the query inputresults in an addition to the KG and the corresponding BC ledger.

It is understood that the query submission may return a response in theform of one or more triplets from the associated KG, as identified by apositive response to the determination at step (820). After theidentified triplets have been processed and sorted (832), the MLMaugments the natural language input to correspond to the sorting of theidentified triplets (834). The augmentation may take one or more forms.For example, in one embodiment, the augmentation arises from a conflictbetween the natural language input and the sorted triplets (836). Whenthe conflict is identified, the augmentation by the MLM is in the formof identification of the correct triplet from the sorting (838), andmodifying the NL input to correspond to the identified triplet (840).The identification at step (838) may take on different forms. Forexample, in one embodiment, the identification may be based on theassociated veracity values, which as described above is a compositescore. Similarly, in one embodiment, one or more of the components thatcomprise the veracity value may be employed as a sorting factor to sortthe list of triplets. In another embodiment, the sort may be based on animmutable factor associated with the triplet entries, sorting thetriplets based on the immutable factor. Accordingly, the augmentationmay be based on an identified conflict.

It is understood that the augmentation may take on other forms inresponse to a match or, in one embodiment, a partial match. When theaugmentation arises from a match between the natural language input andat least one of the triplets in the sorting (842), an entry for thenatural language input is created in the corresponding KG, together withthe BC ledger entry (844). Similarly, when the augmentation arises froma partial match between the natural language input (846) and at leastone of the identified triplets, a new triplet for entry in theassociated KG is created (848). The new triplet corresponds to thereceived NL input, e.g. query submission, and a veracity score isassigned to the new triplet (848). In addition, an entry in a BC ledgercorresponding to the KG is created (850). A BC identifier associatedwith the BC ledger entry is created and stored with the new triplet inthe KG (852) and the assigned veracity score is stored in thecorresponding ledger entry (854). In addition, the new triplet entry inthe KG is connected with triplet(s) identified with the partial match(856). Accordingly, as demonstrated, the augmentation for a match orpartial match includes creation of an entry in the corresponding KG andthe associated BC ledger.

As shown and described in FIGS. 3-8B, the MLM is employed to supportnatural language processing in the form of a query submission toidentify data stored in a KG, and in one embodiment to augment the querysubmission. It is also understood that the MLMs are dynamic and subjectto change. The KG may be utilized to create one or more new MLMs, and/orto retrain an existing MLM. When the ontology is modified, new entitiesand relationships are realized. This new information may then beutilized to automate training of the MLM, thereby support a dynamic andprogressive MLM, create a new MLM, or augment an existing MLM.

Referring to FIG. 9, a flow chart (900) provides a process for trainingan existing MLM. In the process shown herein there is an NLP library ofMLMs. An MLM, referred to herein as a first MLM, within the library isidentified or selected based on its alignment to a knowledge domainexpressed in a KG, referred to herein as a first KG, (902). In responseto receipt of natural language input that is queried against the firstKG, the identified or selected first MLM processes the query input andextracts one or more triplets from the first KG (904). Additionally, asecond KG is identified (906), and in one embodiment is related to thefirst KG. The MLM processes the same query with the second KG and one ormore triplets are extracted from the second KG (908). Each tripletextracted at steps (904) and (908) are also referred to herein asmemories, and include a subject, object, and a relationship. Asdiscussed above, each triplet has an associated BC identifier thatindicates the BC ledger that stores the corresponding veracity value(s).Following step (908), each extracted triplet is processed to identifytheir associated veracity value stored in their corresponding BC ledgerentry (910). The triplet of the first KG and the triplet of the secondKG are evaluated and compared (912). More specifically, the evaluationat step (912) assesses if the content and/or structure of the first KGwas subject to a modification, as reflected in the second KG (914). Fora MLM to be dynamically modified, the determination is if the twosubject KGs have related structure and content. The modification may beevidenced via comparison of the triplets returned from the first andsecond KGs. A negative response to the evaluation at step (914)concludes the MLM modification (922). However, a positive response tothe evaluation at step (914) is followed by identification of thecontent and/or structural change (916). In addition, the correspondingveracity values are assessed to validate the authenticity of thechange(s) (918). Based on the validation at step (918), the structure ofthe MLM is subject to dynamic modification (920).

The modification at step (920) may take different forms. For example, inone embodiment, the modification of the MLM may conform to the validatedchange(s) reflected in the second KG entry as compared to the first KGentry. In another embodiment, the modification may be based on anassessment of the corresponding veracity values of the extracted data.Accordingly, the MLM is demonstrated to be subject to change based uponchanges in the KGs.

Furthermore, it is understood that data and associated relationshipsrepresented in the KGs may be synchronic or diachronic information. Theclassification may be imported into the evaluation at step (912). Datathat should not change and is demonstrated to have been modified shouldnot be reflected in an MLM modification. Accordingly, the dataclassification may be imported into the data evaluation and associatedMLM evaluation.

Referring to FIG. 10, a flow chart (1000) is provided illustrating aprocess for progressive and adaptive MLM configuration. The KG APIperiodically searches an associated or identified KG for new entities,relationships, and data (1002). The identification at step (1002) may beaccomplished by checking the data and/or time of the entries within theKG or comparing the entities and relationships from an existing MLM withdata contained in the KG. A list of entities and relationships that arepresent in the KG and absent from the MLM of interest is produced(1004). The list is produced in a format that is consumable by thetraining tool used to generate MLMs. The consumable data is streamed toupdate the structure of an existing MLM (1006). In one embodiment, theKG API generates a linguistic statement from the KG that expresses eachtriplet that can then be fed to the MLM for training. Following step(1006), the updated MLM is stored in an MLM library as a new MLM (1008).In one embodiment, the progressive MLM configuration is incremental asit represents incremental changes of an existing MLM. Incrementalmachine learning functions to synchronize the MLM with the structure ofa KG. Continuous or incremental changes are performed on a target MLM sothat with each incremental change, the MLM's capability to extract datafrom the KG increases and the MLM effectively adapts.

The system and flow charts shown herein may also be in the form of acomputer program device for use with an intelligent computer platform inorder to facilitate NL processing. The device has program code embodiedtherewith. The program code is executable by a processing unit tosupport the described functionality.

As shown and described, in one embodiment, the processing unit supportsfunctionality to search the corpus for evidence of existing KGs andcorresponding MLMs together with the corresponding BC ledgers andassociated entrie(s). The composite veracity score qualifies and/orquantifies the associated data, and provides a weight for conducting oneor more assessments. The recordation of the veracity score together withthe associated components in a corresponding BC ledger providesauthenticity to the data. Each entry in the outcome set is evaluatedbased on the corresponding veracity score. As described herein, the KGsare subject to modification, including partitioning and linking, as wellas assignment of veracity score components to the data represented orassigned to one or more select KGs. Similarly, as described herein, theMLMs may be dynamically adjusted to reflect structural changes to one ormore of the KGs. More specifically, the MLMs adjust to new entities andentity relationships.

It will be appreciated that there is disclosed herein a system, method,apparatus, and computer program product for dynamic MLM generation andaugmentation through the use of memory and external learning. Asdisclosed, the system, method, apparatus, and computer program productapply NL processing to support the MLM, and for the MLM to support theKG persistence.

While particular embodiments of the present invention have been shownand described, it will be obvious to those skilled in the art that,based upon the teachings herein, changes and modifications may be madewithout departing from this invention and its broader aspects.Therefore, the appended claims are to encompass within their scope allsuch changes and modifications as are within the true spirit and scopeof this invention. Furthermore, it is to be understood that theinvention is solely defined by the appended claims. It will beunderstood by those with skill in the art that if a specific number ofan introduced claim element is intended, such intent will be explicitlyrecited in the claim, and in the absence of such recitation no suchlimitation is present. For non-limiting example, as an aid tounderstanding, the following appended claims contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimelements. However, the use of such phrases should not be construed toimply that the introduction of a claim element by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim element to inventions containing only one such element,even when the same claim includes the introductory phrases “one or more”or “at least one” and indefinite articles such as “a” or “an”; the sameholds true for the use in the claims of definite articles.

The present invention may be a system, a method, and/or a computerprogram product. In addition, selected aspects of the present inventionmay take the form of an entirely hardware embodiment, an entirelysoftware embodiment (including firmware, resident software, micro-code,etc.) or an embodiment combining software and/or hardware aspects thatmay all generally be referred to herein as a “circuit,” “module” or“system.” Furthermore, aspects of the present invention may take theform of computer program product embodied in a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.Thus embodied, the disclosed system, a method, and/or a computer programproduct are operative to improve the functionality and operation of amachine learning model based on veracity values and leveraging BCtechnology.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a dynamic or static random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), a magnetic storage device, a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server or cluster of servers. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

It will be appreciated that, although specific embodiments of theinvention have been described herein for purposes of illustration,various modifications may be made without departing from the spirit andscope of the invention. In particular, the natural language processingmay be carried out by different computing platforms or across multipledevices. Furthermore, the data storage and/or corpus may be localized,remote, or spread across multiple systems. Accordingly, the scope ofprotection of this invention is limited only by the following claims andtheir equivalents.

What is claimed is:
 1. A computer system comprising: an artificialintelligence platform, in communication with a processing unit; aknowledge engine operatively coupled to the processing unit to train amachine learning model (MLM), the knowledge engine configured to:receive natural language (NL) input and query the input against a firstknowledge graph (KG), and extract one or more triplets from the firstKG; apply a selected MLM to a second KG different from the first KG, andextract one or more triplets from the second KG, wherein each tripletincludes a subject, object, and a relationship; for each extractedtriplet: obtain a blockchain (BC) identifier associated with eachtriplet; and identify a triplet veracity value from a corresponding BCledger; detect a modification of the first KG from the extracted one ormore triplets from the second KG; evaluate the detected modification,including employ the obtained BC identifier to assess veracity of thedetected modification; and establish a link between the first KG and thesecond KG, wherein the link creates a relationship between the first KGand the second KG.
 2. The system of claim 1, further comprising theknowledge engine to: identify one or more entities and relationshipspresent in the first KG and absent from the selected MLM; and create theidentified entities and relationships as consumable data.
 3. The systemof claim 2, further comprising the knowledge engine to stream theconsumable data to update the structure of the selected MLM, and storethe updated selected MLM as a new MLM.
 4. The system of claim 1, furthercomprising the knowledge engine to compare the linked first KG andsecond KG, the comparison comprising an evaluation of corresponding oneor more veracity value components.
 5. The system of claim 4, wherein aconflict between two or more data elements in the linked first KG andsecond KG are identified and selectively replaced based on at least oneof the one or more veracity value components.
 6. A computer programproduct to process natural language (NL), the computer program productcomprising a computer readable storage device having program codeembodied therewith, the program code executable by a processing unit to:receive NL input and query the input against a first knowledge graph(KG), and extract one or more triplets from the first KG; apply aselected machine learning model (MLM) to a second KG different from thefirst KG, and extract one or more triplets from the second KG, whereineach triplet includes a subject, object, and a relationship, and foreach extracted triplet: obtain a blockchain (BC) identifier associatedwith each triplet; and identify a triplet veracity value from acorresponding BC ledger; detect a modification of the first KG from theextracted one or more triplets from the second KG; evaluate the detectedmodification, including employ the obtained BC identifier to assessveracity of the detected modification; and establish a link between thefirst KG and the second KG, wherein the link creates a relationshipbetween the first KG and the second KG.
 7. The computer program productof claim 6, further comprising the program code to: identify one or moreentities and relationships present in the first KG and absent from theselected MLM; and create the identified entities and relationships asconsumable data.
 8. The computer program product of claim 7, furthercomprising the program code to stream the consumable data to update thestructure of the selected MLM, and store the updated selected MLM as anew MLM.
 9. The computer program product of claim 6, further comprisingthe program code to compare the linked first KG and second KG, thecomparison comprising an evaluation of corresponding one or moreveracity value components.
 10. The computer program product of claim 9,wherein a conflict between two or more data elements in the linked firstKG and second KG are identified and selectively replaced based on atleast one of the one or more veracity value components.
 11. A method forprocessing natural language (NL), comprising: receiving NL input andquerying the input against a first knowledge graph (KG), and extractingone or more triplets from the first KG; applying a selected machinelearning model (MLM) to a second KG different from the first KG, andextracting one or more triplets from the second KG, wherein each tripletincludes a subject, object, and a relationship, and for each extractedtriplet: obtaining a blockchain (BC) identifier associated with eachtriplet; and identifying a triplet veracity value from a correspondingBC ledger; detecting a modification of the first KG from the extractedone or more triplets from the second KG; evaluating the detectedmodification, including employing the obtained BC identifier to assessveracity of the detected modification; and establishing a link betweenthe first KG and the second KG, wherein the link creates a relationshipbetween the first KG and the second KG.
 12. The method of claim 11,further comprising: identifying one or more entities and relationshipspresent in the first KG and absent from the selected MLM; and creatingthe identified entities and relationships as consumable data.
 13. Themethod of claim 12, further comprising streaming the consumable data toupdate the structure of the selected MLM, and storing the updatedselected MLM as a new MLM.
 14. The method of claim 11, furthercomprising comparing the linked first KG and second KG, the comparisoncomprising an evaluation of corresponding one or more veracity valuecomponents.
 15. The method of claim 14, wherein a conflict between twoor more data elements in the linked first KG and second KG is identifiedand selectively replaced based on at least one of the one or moreveracity value components.
 16. The system of claim 1, wherein the linkmaintains a structure of the first KG and the second KG.
 17. Thecomputer program product of claim 6, wherein the link maintains astructure of the first KG and the second KG.
 18. The method of claim 11,wherein the link maintains a structure of the first KG and the secondKG.